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Abstract 

Research supports small-group emergent literacy intervention to boost preschool 
children’s early skills and provide a solid foundation for continued literacy learning. Although 
such interventions are increasingly available to preschool teachers, we have limited 
understanding as to how these are implemented under routine conditions in authentic classroom 
settings or of the factors that facilitate or hinder implementation. In this study, we examined 
implementation of one small-group emergent literacy intervention, the Nemours BrightStart! 
Program, as used in 36 preschool classrooms (25 schools/centers) whose lead or co-lead teachers 
voluntarily agreed to participate; all classrooms served children identified as being at risk for 
later literacy difficulties. Classrooms were randomly assigned to one of two implementation 
models, with either teachers or other adults (“community aides”) providing the intervention to 
small groups of children identified as at risk for literacy difficulties. Research staff assessed 
implementation using a multi-dimensional framework, with data derived from videos of 
intervention lessons and lesson logs submitted by instructors and instructors’ responses on an 
end-of-year questionnaire. Data were analyzed using quantitative and qualitative approaches. 
Overall, instructors implemented the intervention such that it was delivered with high quality, 
afforded participant responsiveness, and aligned with the intended lesson duration. Adherence 
(i.e., extent to which key lesson elements were present) and the number of lessons implemented 
were more variable, with teachers generally exhibiting better adherence but community aides 
providing more lessons. Factors reported as facilitating implementation pertained to aspects of 
the intervention itself, such as the structured multisensory lessons and their interactive nature, or 


the ability to prepare lessons ahead of time. Time, classroom, and behavior management were 
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commonly reported as challenges. Findings have important implications for intervention 


development, use, and scalability. 
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Implementation of a Small-Group Emergent Literacy Intervention by Preschool Teachers and 


Community Aides 


As encapsulated within emergent literacy frameworks (e.g., Teale & Sulzby, 1987; 
Whitehurst & Lonigan, 1998), literacy development begins well before formal reading and 
writing instruction. Young children’s early understandings about oral and written language 
predict conventional literacy outcomes, and children who enter kindergarten having learned 
emergent literacy skills such as basic print knowledge, phonological awareness, language, and 
emergent writing are well positioned for continued literacy success (National Early Literacy 
Panel, 2008). Yet, for a variety of reasons, many children arrive to kindergarten without these 
competencies (Chatterji, 2006; Zill & West, 2001). These children are at a considerable 
disadvantage, as current research suggests that most children with initially lower emergent 
literacy skills are unlikely to catch up to their more advantaged peers, and many of these children 
will experience literacy difficulties in the later elementary years (Cabell, Justice, Logan, & 
Konold, 2013; Catts, Fey, Zhang, & Tomblin, 2001; Phillips, Lonigan, & Wyatt, 2009). For 
these children, preschool may serve as a means of primary prevention (Dickinson, McCabe, & 
Clark-Chiarelli, 2004) in which targeted support is provided to develop emergent literacy skills 
and provide a solid foundation for continued literacy learning. This targeted support is often 
provided through supplemental interventions delivered in small-group formats. In this study, we 
considered one such supplemental emergent literacy intervention and examined its 
implementation as used by two different types of instructors. 

Small-Group Emergent Literacy Interventions 
Preschool programming often emphasizes children’s language and literacy learning, with 


emergent literacy support integrated across multiple contexts throughout the day (Cabell, 
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DeCoster, LoCasale-Crouch, Hamre, & Pianta, 2013; Early et al., 2010). This general level of 
support, however, may be insufficient for some children who require more intensive emergent 
literacy experiences. Supplemental interventions for supporting these children have proliferated 
over the past 20 years, especially with the rise of response-to-intervention and multi-tiered 
systems of supports approaches to meeting children’s instructional needs (Shepley & Grisham- 
Brown, 2019). These approaches utilize universal screening to identify those children who 
might benefit from more intensive, targeted intervention in a given domain. The targeted 
intervention is offered in a small-group format as a means of differentiating instruction to meet 
individual learning needs (e.g., not all children in a classroom will need supplemental emergent 
literacy support) and increasing the intensity of instruction (e.g., greater instructional time, more 
opportunities for each child to participate/practice; Harn, Linan-Thompson, & Roberts, 2008). 
Successful emergent literacy instruction often features small-group instruction (e.g., see 
Lonigan, Farver, Phillips, & Clancy-Menchetti, 2011; National Early Literacy Panel, 2008; 
Wasik, 2008). Small-group literacy instruction has been found to be more effective than whole- 
class instruction, and equally or more effective while also more efficient than one-to-one 
instruction, in promoting children’s early literacy learning (Ehri et al., 2001; Morrow & Smith, 
1990; Piasta & Wagner, 2010). In the research literature, several small-group interventions have 
demonstrated positive effects on preschool children’s learning of emergent literacy skills. For 
example, Sound Foundations, a small-group phonemic awareness intervention, produces initial 
increases in children’s phonemic awareness that translate to better decoding skills in elementary 
school (Byrme & Fielding-Barnsley, 1991, 1995). PAth to Literacy, which couples phonological 
awareness and alphabet knowledge intervention, shows positive effects on phonological 


awareness (Goldstein et al., 2017; Kruse, Spencer, Olszewski, & Goldstein, 2015), and the 
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Nemours BrightStart! program, which intervenes on multiple emergent literacy skills, improves 
phonological awareness and print knowledge (Bailet, Repper, Murphy, Piasta, & Zettler-Greeley, 
2013; Bailet, Repper, Piasta, & Murphy, 2009; Zettler-Greeley, Bailet, Murphy, DeLucca, & 
Branum-Martin, 2018). Several small-group interventions have been shown to improve 
vocabulary learning, including World of Words (Neuman & Kaefer, 2013; see also Neuman, 
Newman, & Dwyer, 2011), Words of Oral Reading and Language Development (Gonzalez et al., 
2011; Pollard-Durodola et al., 2016), and Story Friends (Kelley, Goldstein, Spencer, & Sherman, 
2015; E. J. Spencer et al., 2012), with the latter also having modest effects on listening 
comprehension. Additional small-group interventions, such as Language in Motion and Story 
Champs, have targeted and improved other aspects of children’s language development (Phillips 
et al., 2016; T. D. Spencer, Petersen, & Adams, 2015; T. D. Spencer & Slocum, 2010). These 
intervention effects have been established using rigorous, experimental designs generally in 
comparison to typical, Tier | instruction (cf. Byrne & Fielding-Barnsley, 1991; Goldstein et al., 
2017; Neuman & Kaefer, 2013). Given their promise, these types of small-group emergent 
literacy interventions are now becoming commercially available and marketed directly to 
preschool teachers and administrators. 
Implementation of Small-Group Interventions in Preschool 

Despite the rise and promise of small-group interventions for promoting emergent 
literacy development during preschool, there is limited information concerning implementation 
of such interventions by preschool teachers. Implementation refers to how the intervention is 
used in practice and, in turn, affects the extent to which interventions achieve intended outcomes 
(Durlak, 2010). Implementation is thus the mechanism through which an intervention produces 


effects; variation in how core intervention components are implemented will result in variation in 
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outcomes. Because of this, it is critically important to understand how interventions are 
implemented by their intended end users, such as teachers (Gottfredson et al., 2015). 

Implementation can be characterized across multiple dimensions (Dane & Schneider, 
1998; Durlak, 2010; Durlak & DuPre, 2008; Dusenbury, Brannigan, Falco, & Hansen, 2003). 
Often, implementation is described in terms of adherence, or the extent to which key elements of 
the intervention are delivered as described in program materials, and exposure or dosage, 
referring to how much of the intervention was provided (e.g., number of sessions, duration of 
sessions). These two dimensions of implementation are commonly referred to as fidelity or 
treatment integrity in the literature. Implementation can also be characterized in terms of quality 
of delivery, or how well the intervention is conducted, and participant responsiveness, or the 
extent to which recipients (e.g., children) are engaged during the intervention. Additional 
dimensions include the uniqueness of the intervention from other interventions or practices, 
differentiation from what children would otherwise experience in the absence of the intervention, 
the extent to which the intervention reaches those in need of services, and any adaptations or 
modifications made to the intervention. All of these dimensions are influenced by ecological 
factors including those related to perceived social validity of the intervention (e.g., acceptability, 
feasibility, fit) as well as those related to the context in which the intervention is being used (e.g., 
leadership, resources, and other organizational supports; Durlak & DuPre, 2008). 

To date, most studies of small-group emergent literacy interventions have relied on 
research staff as instructors (e.g., Byrne & Fielding-Barnsley, 1991; Kruse et al., 2015; Lonigan 
& Phillips, 2016; Neuman & Kaefer, 2013; Phillips et al., 2016; T. D. Spencer et al., 2015; 
Zettler-Greeley et al., 2018). This is important for ensuring that the intervention is delivered as 


intended and affords strong claims regarding efficacy. However, given that these interventions 
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are ultimately intended for use by practitioners, it is also necessary to understand implementation 
when used by teachers in authentic preschool classrooms. This is particularly true for 
interventions that utilize a small-group format. Small-group instruction is not prevalent in 
preschool classrooms and may be challenging for teachers to implement (Bratsch-Hines, 
Burchinal, Peisner-Feinberg, & Franco, 2019; Connor, Morrison, & Slominski, 2006; Farley, 
Piasta, Dogucu, & O’Connell, 2017; Sawyer et al., 2018). Farley and colleagues, for instance, 
found that most teachers in their sample provided less than 9 min a day of small-group emergent 
literacy instruction with a mode of 0; the likelihood of using small-group instruction was greater 
in the presence of better classroom management, better instructional support, and lower 
adult:child ratios. Others have similarly acknowledged the importance of effective classroom 
routines and management to support small-group instruction, along with the difficulties that 
teachers often encounter when trying to organize and implement such instruction in their 
classrooms (AI Otaiba et al., 2011; Kaminski, Powell-Smith, Hommel, McMahon, & Aguayo, 
2014; Wasik, 2008; Zucker, Solari, Landry, & Swank, 2013). Small groups may be especially 
challenging in preschool contexts, where many teachers may not have formal classroom 
management training, children are still learning behavioral expectations for the classroom 
environment, and aides or assistants may not be available. In the handful of studies involving 
preschool teachers as instructors for small-group emergent literacy interventions, implementation 
has been supported in ways that help overcome these issues (cf. Goldstein et al., 2017). For 
instance, in studies of Words of Oral Reading and Language Development, all participating 
teachers were located in public school-based classrooms and had regular access to in-classroom 
aides (Gonzalez et al., 2011; Pollard-Durodola et al., 2011). In a study of small-group language 


intervention by Zucker et al. (2013), all teachers experienced over 100 hours of professional 
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development, including classroom management training, prior to implementation. However, 
such supports may not be typically afforded to preschool teachers, especially those who work 
outside the public school system (estimated at 94%; National Survey of Early Care and 
Education, 2013). 

Recognizing these challenges and the many demands on preschool teachers, some have 
tested less conventional means for providing small-group emergent literacy instruction within 
preschool classrooms. Story Friends, for example, automates the intervention; children 
experience the intervention via prerecorded stories and embedded lessons at a classroom 
listening center (Kelley et al., 2015), which minimizes small-group management and frees the 
teacher to work with the rest of the class. In a few cases, volunteers or community members 
have been leveraged to provide supplemental, one-to-one or small-group emergent literacy 
intervention (Fitzgerald, Robillard, & O’Grady, 2018; Markovitz, Hernandez, Hedberg, & 
Silbergelitt, 2015; Nelson, Sanders, & Gonzalez, 2009), drawing from studies supporting 
positive effects of such programs for elementary-aged children (for review, see Ritter, Barnett, 
Denny, & Albin, 2009). In these studies, adults other than the classroom teacher serve as the 
instructor and provide intervention to identified children, again freeing the teacher to work with 
the rest of the class. In work by Nelson and colleagues (2009), for example, community 
members were recruited by Head Start centers to provide small-group phonological awareness 
and alphabet knowledge intervention; although the study authors did not report data concerning 
implementation, the intervention showed positive, direct effects on children’s alphabet 
knowledge learning and potential positive effects on phonological awareness outcomes. The 
Nemours BrightStart! program resulted from a clinic-community partnership between Nemours 


Children’s Health System in Jacksonville, Florida and local preschool centers; it was developed 
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to address growing concerns regarding literacy difficulties. Nemours hired former teachers to 
visit partnering preschools and provide the small-group emergent literacy intervention, at no 
cost, to children identified as being at risk for later literacy difficulties; these teachers also 
assisted with program and lesson development. Studies indicate that the program has been 
implemented with high adherence and generally anticipated dosage, and, as noted previously, 
showed positive effects on children’s phonological awareness and print knowledge (Bailet et al., 
2009, 2013; Zettler-Greeley et al., 2018), thereby supporting the program’s underlying logic 
model (Figure 1). 
The Present Study 

As we have argued, a growing research base supports the use of small-group 
interventions for promoting preschool children’s emergent literacy development. Realizing the 
promise of such interventions, however, is dependent on how such interventions are 
implemented when used under routine conditions in authentic preschool classrooms. Thus, in 
the present study, we examined the implementation of one small-group emergent literacy 
intervention, namely the Nemours BrightStart! program (NBS!), as used directly by preschool 
teachers and within a community-implementation context. We selected NBS! because it has 
prior evidence of positive effects on children’s outcomes, is now commercially available for 
preschool teachers to use, and previously has been used in a community partnership. With 
respect to the latter, NBS! includes many features that are considered necessary for successful 
implementation of literacy programs by non-teachers (e.g., structured lessons, provision of high- 
quality materials, availability of training/professional development; Wasik, 1998). Some of 
these features are included in the NBS! logic model (Figure 1) specifying its core components. 


As the logic model depicts, it is the implementation of these core components that is 
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hypothesized to drive the improvements in early literacy skills noted in previous efficacy studies 
(Bailet et al., 2009, 2013; Zettler-Greeley et al., 2018). As such, it is important to examine 
implementation beyond the original community-clinic partnership, to determine feasibility for 
scale up (Gottfredson et al., 2015). In the present study, we examine implementation of NBS! as 
used by preschool classroom teachers and in a new community context when implemented by 
adults (“community aides”) affiliated with a local kindergarten readiness initiative. 

We expected that implementation might differ between the two types of instructors. On 
one hand, implementing a literacy intervention with high adherence and quality of delivery may 
require specialized content and pedagogical knowledge (Vadasy, Jenkins, Antil, Wayne, & 
O'Connor, 1997), which may be more characteristic of preschool teachers. On the other, small- 
group intervention requires considerable classroom management and may be more easily 
implemented with another adult present (Farley et al., 2017). We also anticipated that the two 
types of instructors might have different perceptions and experiences implementing the program, 
although we did not have specific expectations regarding this. We addressed the following four 
research questions: 

(1) How was the NBS! program implemented by instructors, with respect to adherence, 
dosage, quality, and participant responsiveness? 

(2) Did implementation differ when the NBS! program was used by classroom teachers 
versus community aides? 

(3) To what extent did the two types of instructors view the NBS! program as socially valid? 

(4) What did instructors identify as challenges, facilitators, or possible adaptations for 


implementing the NBS! program? 
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Method 

The current study involved the first two cohorts of teachers and community aides who 
implemented Nemours BrightStart!: The Complete Program for Early Literacy Success, Level 
One, as part of a larger project. We recruited 58 preschool classrooms that served children 3 to 5 
years old in which the lead/co-lead teacher voluntarily provided informed consent, instructed at 
least one child identified as at risk for literacy difficulties based on the Get Ready to Read!- 
Revised early literacy screener (Whitehurst & Lonigan, 2010), and committed to all study 
activities. We used a random number generator to randomly assign classrooms to one of three 
conditions: NBS! teacher-implemented, NBS! community aide-implemented, or business-as- 
usual control. Thirty-nine classrooms were assigned to one of the two NBS! conditions, in 
which one small group of children received the NBS! intervention. Prior to beginning NBS! 
implementation, teachers in two classrooms withdrew from the study, and a third classroom 
became ineligible because all children identified to participate in the intervention stopped 
attending preschool. Thus, 36 classrooms, located in 25 different schools/centers, contributed 
data to this study. 

Classrooms. The majority of teachers in these 36 classrooms were female (94%). Forty- 
seven percent were Black, and 53% were White; one teacher (3%) was Hispanic/Latinx. 
Teaching experience ranged from 0 to 30 years (IM = 13.41), with most of those at the preschool 
level (M = 12.03; 2 unreported). Fourteen percent held teaching licenses. Four teachers (11%) 
held high school diplomas as the highest degree earned; 31% had an associate’s degree, 47% had 
a bachelor’s degree, and 6% had a master’s degree (6% unreported). Most classrooms were 
situated in early childhood centers (81%) or public schools (14%) located in urban (81%) or 


suburban (19%) settings. Thirty-six percent were affiliated with Head Start, and 50% accepted 
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public subsidies (8% unreported). Eleven percent of classrooms were half-day only, 67% were 
full-day programs, and 17% provide both half- and full-day options (6% unreported). Class sizes 
ranged from 7 to 20 children (M = 13.85), with an average teacher:child ratio of 1:6.34. Ten 
teachers (four and six in the teacher-implemented and community aide-implemented conditions, 
respectively) reported having other adults regularly visit and help within the classroom. Thirty- 
three percent of classrooms enrolled at least one child who had an individualized education plan 
(range = 1 to 3; M= 1.33). Fifty-eight percent enrolled at least one child who was a dual/multi- 
language learner (range = | to 8; M@ = 3.43). Creative Curriculum was the most frequently 
reported curriculum (81%); other reported curricula included Assessment, Evaluation, and 
Programming System (8%), Handwriting Without Tears (8%), Opening the World of Learning 
(6%), Reggio Emilia Approach (8%), and Scholastic (11%). 

Descriptive information for these 36 classrooms, split by condition assignment, is 
presented in Table 1. Seventeen were randomly assigned to the teacher-implemented condition, 
and 19 were randomly assigned to the community aide-implemented condition. Because 
classrooms constituted the unit of analysis for our first two research questions, we compared 
classrooms across these characteristics using Chi-square test, Fisher’s Exact Test, or one-way 
ANOVA as appropriate to the categorical or continuous data. Classrooms were similar on all 
characteristics with one exception: By chance, classrooms assigned to the community-aide 
implemented condition had significantly better classroom organization than those assigned to the 
teacher-implemented condition, F(1,31) = 4.78, p = .036. 

Community aides. For the 19 of 36 classrooms that were randomly assigned to the NBS! 
community aide-implemented condition, adults affiliated with a local kindergarten readiness 


initiative, referred to as “community aides,” visited classrooms to implement NBS!. This 
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initiative began in 2013; it partners with local early childhood programs to provide emergent 
literacy screenings and literacy-focused professional development. Nine community aides 
originally participated in the project but two withdrew for personal reasons after implementing 5 
to 6 lessons; their classrooms were reassigned to other community aides. All but one community 
aide implemented in multiple classrooms (range of 2 to 5), and three community aides 
implemented for both of the two cohorts reported in this study (i.e., implemented for two 
sequential years in different classrooms). Community aides were selected, scheduled, and 
managed by the initiative staff; no criteria were set beyond an interest in working with the 
initiative, voluntarily providing informed consent, and successfully completing a background 
check. When asked why they were interested in working with the initiative, all community aides 
cited an interest in supporting education and/or early literacy within the community; two cited 
their enjoyment of working with children and previous positive experiences serving as tutors or 
volunteers. All community aides received hourly compensation for their initiative activities. 

All seven community aides were female. Fifty-seven percent were Black, 29% were 
White, and one (14%) was multiracial. One (14%) had some college but no degree, two (29%) 
had a bachelor’s degree, three had a master’s degree (43%), and one had a doctoral degree 
(14%); most degrees were not in the field of education, although one had a degree in early 
childhood education and another in agricultural extension education. All had previous experience 
working with children. Two (29%) had prior classroom teaching experience: One substitute 
taught in elementary classrooms for | year, and the other had 6 years experience as a lead 
preschool teacher plus 2 years experience as a preschool aide. The latter community aide held a 
teaching license. Four community aides (57%) had prior experience as reading tutors through 


AmeriCorps, a statewide mentoring program, or a city/library partnership. Three community 
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aides (43%) had experience with one or more educational curricula; these included Handwriting 
Without Tears, Let’s Begin with the Letter People, Scholastic, Houghton Mifflin, and Orton- 
Gillingham. Other experiences included serving as a Sunday School teacher, classroom 
volunteer, children’s choir director, and summer camp aide. 
NBS! and Professional Development 

NBS! is a research-based, small-group supplemental intervention designed to support the 
emergent literacy skill development of children identified as at risk for literacy difficulties (for 
additional information, see Bailet et al., 2009, 2013; Zettler-Greeley et al., 2018). NBS! 
provides systematic instruction to support print knowledge, phonological awareness, language 
and comprehension, and emergent writing via a set of 20 scripted lessons that follow a specified 
scope and sequence. Lessons follow a familiar, seven-part routine. The Opening Activity 
engages children in the lesson and introduces/reviews key print knowledge and phonological 
awareness concepts. The Letter Introduction provides instruction in letter names and sounds with 
one new letter introduced per lesson. The Read Aloud provides opportunities for rich language 
and print interactions in the context of reading high-quality, authentic children’s books; research- 
based strategies are embedded into the read aloud (e.g., print referencing, dialogic questioning, 
embedded vocabulary instruction, and comprehension-facilitating techniques) to meet learning 
objectives. Core Activities provide carefully sequenced instruction on new skills, primarily 
emphasizing print knowledge and phonological awareness. The Emergent Writing component 
engages children in the writing process, including pre-writing planning and post-writing sharing, 
as well as name writing and letter formation. The Wrap Up involves a final activity that reviews 
and integrates content taught throughout the current and previous lessons. In the Reflection, the 


instructor briefly dialogues with children to review the lesson content and provide specific 
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positive feedback; the instructor also sends home a Family Letter that describes lesson content 
and suggestions for how caregivers can build on lesson content at home. Throughout the lesson, 
instruction follows a gradual release of responsibility model (Pearson & Gallagher, 1983) in 
which concepts are explicitly taught and modeled, practiced with instructor support, and 
practiced independently. Lessons are multisensory, such that all involve visual, auditory, 
kinesthetic, and tactile components. All lessons are also accompanied by strategies to adapt or 
accelerate instruction to meet individual learning needs and extension activities. Lessons can be 
implemented as one, 50-min session or split into two 20-30 min sessions. 

Teachers assigned to the NBS! teacher-implemented condition and community aides 
assigned to the NBS! community-aide implemented condition received all materials for 
implementing NBS!, including the instructor guide, lesson plans, and all materials (e.g., books, 
manipulatives), and associated professional development in October during their year of 
participation. Professional development mirrored what would be typically provided to teachers 
who purchase the NBS! program through the publisher and opt for a 2-day face-to-face training; 
teachers and community aides participated together in these full-day trainings. The professional 
development included an overview of the program and its instructional design, step-by-step 
training in all lesson components, demonstrations and video exemplars of implementation, and 
considerable hands-on practice. For purposes of the research study, instructors also received 
instructions for video recording lessons and completing lesson logs as part of professional 
development, which were used to measure implementation as described further below. 

Upon completing professional development, the teacher or community aide, as assigned, 
began implementing NBS! with one small group of children identified as at risk for literacy 


difficulties. Children were identified by the research team via the widely used Get Ready to 


IMPLEMENTATION OF SMALL-GROUP INTERVENTION 19 


Read!-Revised early literacy screener, and up to four children who scored below age-based cut 
scores were selected for intervention in each classroom; small groups averaged 3.08 children (SD 
= 1.11). Instructors were asked to implement the 20 NBS! lessons at a rate of one lesson per 
week, split into two 20-30 min sessions; individual instructors selected the days of the week on 
which to provide the two sessions. In the case of child absences, instructors were asked to 
continue lessons with children in attendance and to make up missed lessons as feasible. 
Measures 

We measured implementation following the multi-dimensional framework set forth by 
Durlak and colleagues (Durlak, 2010; Durlak & DuPre, 2008) and also considered instructors’ 
perceptions concerning social validity, facilitators, and challenges of using NBS! given that such 
ecological factors may influence implementation (Briesch, Chafouleas, Neugebauer, & Riley- 
Tillman, 2013; Durlak, 2010). Data concerning implementation were derived from three 
sources: (1) video recordings of lessons, (2) lesson logs completed by instructors throughout 
implementation, and (3) an end-of-year questionnaire. With respect to video recordings, we 
asked instructors to videotape all lessons that they implemented. The research team coded 
lessons | and 2 (as a means of providing immediate feedback on lesson implementation) and a 
randomly selected 50% of the subsequent lessons using two coding schemes described below. 
Coders had no direct interaction with any participants and were not told whether videotaped 
lessons were conducted by teachers or community aides, in an effort to keep them blind to NBS! 
condition. Coders followed a rigorous training protocol and met an initial criterion of 90% exact 
agreement with master-coded lessons. Double coding of a randomly selected 20% of video 


recorded lessons indicated high interrater reliability (intraclass correlations [ICCs] of .96 for the 
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Nemours Fidelity Implementation Record and .82 for the Quality of Intervention Delivery and 
Receipt tool). 

Adherence. We measured adherence by coding selected videos using the Nemours 
Fidelity Implementation Record (Nemours BrightStart!, 2016). The Nemours Fidelity 
Implementation Record is a fidelity checklist that captures the extent to which instructors 
adhered to key elements of NBS! lessons, with respect to the environment (e.g., lesson space 
allowed for movement), organization (e.g., materials were gathered in advance), general delivery 
methods (e.g., instructor used identified multisensory teaching strategies), and specific 
instructional practices for each part of the lesson routine (e.g., Letter Introduction: instructor 
modeled letter formation; Read Aloud: instructor asked children for predictions; Cronbach’s a = 
.88). Each element was rated as to whether present (1 point) or absent (0 points). Because the 
total number of elements varied by lesson depending on its specific content and activities (range 
of 55 to 76), adherence was calculated as the proportion of elements implemented per coded 
lesson. For analyses, we calculated the average adherence across coded lessons for each small 
group. 

Dosage. We measured dosage both in terms of lesson duration and the number of lessons 
implemented. We calculated lesson duration, in min, from selected videos (summing across the 
two sessions per week) and derived the average lesson duration per small group. Instructors 
reported each lesson implemented via logs, and we calculated the total number implemented 
(maximum of 20). 

Quality. We measured the quality with which instructors implemented NBS! by coding 
selected videos using the Quality of Intervention Delivery and Receipt tool (Harn, Forbes-Spear, 


Fritz, & Berg, 2012). This tool has demonstrated adequate interrater reliability (e.g., ICC = .71), 
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consistency (correlations over time of .75 to .95), and predictive associations with other 
instructional measures (e.g., Classroom Assessment Scoring System) and children’s learning 
outcomes (Fritz, Harn, Biancarosa, Lucero, & Flannery, 2019). The Quality of Intervention 
Delivery portion of the tool assesses the process quality with which an intervention is delivered; 
items reflect principles of high-quality instruction as grounded in the empirical literature (e.g., 
using clear and consistent wording, frequent modeling, positive reinforcement; see Fritz et al., 
2019) and are rated on a 0 to 3 scale: 0 = lack of implementation, 1 = inconsistent 
implementation, 2 = effective implementation, and 3 = expert implementation. We omitted two 
Quality of Intervention Delivery items that were not applicable to NBS! (pertaining to 
corrections for incorrect responses and ensuring mastery before moving to the next lesson). We 
calculated the average across each of the remaining 13 items for each coded lesson, and then 
calculated the average quality across coded lessons for each small group (Cronbach’s a = .85). 
Participant responsiveness. We used both the Nemours Fidelity Implementation 
Record and the Quality of Intervention Delivery and Receipt tool to measure participant 
responsiveness. The Nemours Fidelity Implementation Record includes four items pertaining to 
child attention and participation during lessons (e.g., each child had an opportunity to use the 
NBS! materials). Each item was rated as present (1 point) or absent (0 points). We calculated 
the average across the items for each coded lesson, and then calculated the average across 
lessons for each small group (Cronbach’s a = .36, likely due to limited variability, as discussed 
in Results, and few items). The Quality of Intervention Delivery and Receipt tool includes four 
items pertaining to children’s responses to intervention (e.g., behavioral and emotional 
engagement, following routines and directions), which are scored on a 0 to 3 scale: 0 = no or 


only one child is responding, 1 = some children are responding, 2 = most children are 
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responding, and 3 = all children are responding. We calculated the average across these items for 
each coded lesson and then the average across lessons for each small group (Cronbach’s a = .83). 

Social validity. We measured the social validity of NBS! using an adapted version of the 
Usage Rating Profile-Intervention (Briesch et al., 2013) which was included on our end-of-year 
questionnaire. The Usage Rating Profile-Intervention assesses multiple aspects of social validity 
that may affect use of an intervention by practitioners, including acceptability, understanding, 
feasibility, and fit/support within the context (e.g., preschool center). We modified the Usage 
Rating Profile-Intervention by removing items pertaining to family-school connections, as these 
were not relevant to NBS!, and specifying “NBS!” rather than the more general “intervention” in 
item wording. We also did not ask community aides to respond to items that were irrelevant to 
them (one concerning fit with current instructional practices and five pertaining to fit within the 
preschool center); teachers thus responded to 26 items and community aides responded to 20 
items using a 0 to 5 scale on which higher scores reflect more positive perceptions of social 
validity. We calculated the average rating across all items (Cronbach’s a = .93) for each 
instructor; for the three community aides who implemented for both of the two cohorts, we only 
considered their reported social validity in their first year of implementation. 

Facilitators, challenges, and adaptations. On the end-of-year questionnaire, we asked 
instructors to respond to four open-ended questions: (1) What did you like about NBS!? (2) 
What, if any, challenges did you face in using NBS!? In particular, if you were not able to 
complete all 20 lessons of NBS!, what challenges made completing the lessons difficult? (3) Do 
you have any feedback for the developers of NBS!? (4) Do you have any suggestions for future 
instructors who choose to use NBS!? Research team members used content analysis (Hsieh & 


Shannon, 2005) to qualitatively code these four open-ended questions and identify facilitators, 
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challenges, or suggested adaptations noted by teachers and community aides. Each open-ended 
question was answered by at least 87% of instructors, indicating that responses were generally 
representative of the sample as a whole. 

Two research team members reviewed and coded instructors’ responses using a multi- 
step process. First, both researchers independently reviewed a subset of responses and generated 
a list of broad codes that characterized the content of responses (lesson structure, engagement, 
time management, classroom management, behavior management). Next, the researchers 
engaged in an iterative process to confirm and elaborate codes as they coded the full corpus of 
responses. This iterative process involved (a) individually coding responses and noting subcodes 
(i.e., more specific codes within each broad code; note that no additional broad codes emerged), 
(b) jointly discussing emerging findings to achieve consensus and revise the coding scheme, and 
(c) independently applying the revised coding scheme to all responses coded thus far. Multiple 
codes could be applied to single responses. For example, the response, ‘...to allow teachers to 
have more room for making the lessons unique...there were times when my students had grasped 
the concept and the constant repeating of a concept seemed to bore them,’ was coded as related 
to both the themes of lesson structure and engagement. The iterative coding process was 
repeated until the researchers agreed that no further revisions to the coding scheme were 
necessary. The researchers then independently reviewed all coded responses to assign valence 
codes, which signaled whether the response indicated a facilitator or challenge (e.g. in the 
example above, references to lesson structure and engagement were both coded as challenges); 
suggested adaptations were also noted. Consensus again was achieved through discussion. 
Finally, researchers reviewed the final, coded responses to interpret findings. Table 2 presents 


the final codes for reported facilitators and challenges. 
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Results 

Implementation and Social Validity 

Table 3 presents descriptive data across all instructors for adherence, dosage, quality of 
delivery, and participant responsiveness, which addresses our first research question; correlations 
among implementation measures are presented in Table 4. On average, instructors implemented 
62% of lessons (12 of 20) with 76% adherence. Notably, there was considerable variability 
among small groups across most implementation measures. One teacher, for instance, did not 
implement any lessons; although she did not withdraw from the study, she indicated that, despite 
her intention to implement the NBS! program, she did not have time to complete the small-group 
lessons. Across the small groups, five (14%) received less than 5 lessons, seven (19%) received 
between 5 to 9 lessons, six (17%) received 10 to 14 lessons, 13 (36%) received 15 to 19 lessons, 
and five (14%) received all 20 lessons. Adherence ranged from 48% to 97% of key lesson 
elements implemented; adherence averaged less than 65% for five small groups (14%), between 
66% to 75% for 7 small groups (20%), between 76% to 85% for 15 small groups (43%), between 
86% to 95% for 7 small groups (20%), and over 95% for 1 small group (3%). Average lesson 
duration was 42.50 min, with lessons for 17 small groups (47%) falling within the intended range 
of 40 to 60 min; fourteen small groups (39%) tended to experience lessons of shorter duration, 
and four small groups (11%) tended to experience lessons of longer duration. In general, 
instructors’ quality of delivery was rated as “effective.” The average quality ratings for the 
majority of small groups (22 small groups; 62%) indicated between “effective” and “expert” 
implementation; average quality ratings for 13 small groups (36%) were between “inconsistent” 
and “effective” implementation, and no quality ratings averaged below 1.53, which would have 


indicated “inconsistent” or lack of implementation. Children were rated as engaging and 
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participating at high levels during lessons on both participant responsiveness measures. Ratings 
on the relevant items from the Nemours Fidelity Implementation Record were positively skewed: 
Only 5 small groups (14%) experienced lessons averaging less than 88% on these items, and half 
(18 small groups, 50%) averaged 100%. On the relevant items from the Quality of Intervention 
Delivery and Receipt tool, ratings indicated that “most” children were responding to lessons, 
with this being typical for 30 small groups (84%). Three small groups (8%) experienced lessons 
with average ratings indicating that between “some” and “most’ children were responding, and 
two (6%) experienced lessons in which “all” children were consistently responding. Descriptive 
data for social validity, by instructor, are also presented in Table 3. Instructors generally agreed 
with statements about the social validity of NBS! (M = 3.68); one teacher and one community 
aide tended to disagree with these statements (i.e., ratings below 3). 

Table 3 also presents implementation data for teachers versus community aides to address 
our second research question. Given that community aides served as instructors for more than 
one small group, we conducted preliminary analyses to determine the extent of shared variance 
due to instructor. Unconditional ICCs, nesting small groups within community aide, indicated 
substantial variance due to community aide; ICCs were largest for adherence (.91) and quality of 
delivery (.73), followed by dosage (.56 and .47 for lesson duration and number of lessons 
implemented, respectively), and smallest for participant responsiveness (.12 and .21 as rated on 
the Nemours Fidelity Implementation Record and Quality of Intervention Delivery and Receipt 
tool). We thus compared teacher versus community aide implementation via multilevel models, 
which is the more conservative approach. We used SAS proc mixed to estimate multilevel 
models for all variables except social validity, which was analyzed at the instructor rather than 


small group level. Notably, the pattern of results was the same when using traditional, single- 
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level approaches and robust estimation methods. We also calculated effect sizes to aide 
interpretation of practical significance (i.e., whether differences between instructors were 
meaningful). To do so, we divided the estimated difference between instructors based on the 
statistical models by the pooled standard deviation. Thus, we considered differences in 
implementation in terms of both statistical and practical significance, following Fan (2001). 
Statistical results and effects sizes are presented in Table 3. Results indicated statistically 
significant differences between teacher and community aide implementation only in terms of the 
number of lessons implemented. Community aides implemented approximately seven more 
lessons than teachers, which we considered a practically meaningful difference and which 
corresponded to a large effect size (d = -1.17). We also noted large, although not statistically 
significant, differences in adherence (d = 0.70), participant responsiveness as measured by 
relevant items from the Quality of Intervention Delivery and Receipt tool (d = 0.51), and social 
validity (d = 0.57) and considered whether any of these differences were practically significant. 
With respect to adherence, teachers implemented 81% of key lesson elements, on average, 
whereas community aides implemented 73% of key lesson elements. We considered this to be a 
meaningful difference based on the developers’ expectation that lessons are implemented with at 
least 75% adherence. Adherence was below this criterion for only three small groups taught by 
teachers (19% of small groups in the teacher-implemented condition) but nine small groups 
taught by community aides (47% of small groups in the community aide-implemented 
condition). With respect to participant responsiveness via the Quality of Intervention Delivery 
and Receipt tool, only two small groups were consistently rated as having “all” children 
responding and both were taught by teachers. We considered this to be practically meaningful, 


despite similarities across conditions in the percentages of small groups rated as between “all” 
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and “most” children responding (81% of those taught by teachers, 89% of those taught by 
community aides) and rated as between “some” and “most” children responding (6% of those 
taught by teachers, 11% of those taught by community aides). Finally, with respect to our third 
research question concerning social validity, the large effect size favoring teachers corresponded 
to a difference of only four-tenths of a point. Descriptively, eight teachers (47%) but only one 
community aide (14%) averaged between “agree” and “strongly agree” on the social validity 
statements, eight teachers (47%) and five community aides (71%) averaged between “somewhat 
agree” and “agree,” and one teacher (6%) and one community aide (14%) averaged between 
“somewhat disagree” and “somewhat agree.” Thus, teachers tended to view NBS! more 
favorably, but it is unclear whether this represents a meaningful difference. In summary, we 
found statistically and practically significant differences in the number of lessons implemented 
favoring community aides, and practically significance differences in adherence and participant 
responsiveness favoring teachers. 
Facilitators, Challenges, and Suggested Adaptations 

For our fourth and final research question, we qualitatively documented the facilitators 
and challenges reported by teachers and community aides (see Table 2). All but one instructor 
identified at least one facilitator and one challenge to implementation. The most commonly 
reported facilitators pertained to lesson structure, engagement, and time management. Many 
instructors cited the provision of materials and multisensory activities (n = 20; 13 teachers, 7 
community aides) and scripted lessons (” = 7; 1 teacher, 6 community aides) as facilitators. For 
example, one teacher reported that she “liked the method in which the lessons were presented to 
the children. It kept their attention by using props and games to convey the lessons. I like that 


instructors had two opportunities to present materials in each lesson before moving on and how 
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the dialogue was provided for the teachers to say to their students.” Other instructors noted 
practice and preparation ahead of time (” = 7; 4 teachers, 3 community aides) and the interactive 
nature of the lessons (n = 8; 2 teachers, 6 community aides) as facilitators. As another teacher 
reported, it “helps to practice the lessons before implementing them with students,” and she liked 
how the lessons “keep kids active and has something for each child’s way of learning.” Many 
instructors commented on children’s engagement with the lessons, which was seen as promoting 
child interest and learning, and attributed this to aspects of the lessons themselves. For example, 
one teacher responded, “I think most of lessons were fun and engaging for children. The same 
basic pattern in the lessons made it easier for them to expect what was next. The materials were 
well made and the books chosen, for the most part, were fun and interesting.” Similarly, other 
instructors commented that “the books were great and the kids really liked them” and “I liked the 
very interactive lessons and having a close learning relationship with a small group.” 

The most commonly reported challenges to implementation pertained to time 
management, classroom management, and behavior management. These challenges included 
limited time or difficulty incorporating lessons into daily routines (” = 15; 10 teachers, 5 
community aides) and lack of staff support (7 = 10; all teachers). As one teacher reported, “‘it is 
a great program but hard for lead teachers to leave the room and to not incorporate all children.” 
Another teacher reported, “I was never able to leave the classroom to get set up or take the 
children out. Little staff support meant I had less time to be out of the room.” Community aides 
reported particular challenges in scheduling with teachers/centers/schools (n = 4; all community 
aides) and inability to complete lessons due to child absences (n = 13; 6 teachers, 7 community 
aides). As a community aide noted, “Students missing many lessons was a challenge. Even 


when attempts were made to give a make-up lesson the students are still absent...Trying to 
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manage two schools and coordinate between schedules was hard.” Behavior issues during small 
groups was also a commonly reported challenge for community aides (7 = 5; 1 teacher, 4 
community aides); as one reported: “I had bouncy, energetic children that I had to redirect back 
to the lessons. One lesson I had to remove a child because his behavior was disruptive to the 
lesson.” Interestingly, although both teachers and community aides reported lesson structure as a 
facilitator to lesson implementation, repetitiveness and difficulty of lesson content were also 
reported as challenges to implementation (n = 8; 5 teachers, 3 community aides). For instance, 
one community aide reported that “The activities did not seem appropriate for the intellectual 
and academic capabilities of the students. They were way too advanced. We would spend an 
entire lesson talking about various activities and at the end the kids still didn’t know the letter 
and sound.” This indicates that particular aspects of NBS! were viewed as both facilitators and 
challenges to implementation. 

In addition to facilitators and challenges, 10 instructors suggested potential adaptations to 
consider for future lesson implementation. These were often reported in light of the challenges 
that instructors faced, and the most common suggestions were adaptations pertaining to lesson 
structure, classroom management, and behavior management. Instructors recommended allowing 
more flexibility in lesson structure in order to make lessons unique to the needs of the children ( 
= 5; 3 teachers, 2 community aides), having instructors other than the lead teacher implement 
lessons in the future (n = 3; all teachers), ensuring there is enough staff to support instructors 
during lesson implementation (n = 2; both teachers), and learning more about small group 
instruction (e.g. addressing behavior challenges during small groups, utilizing a designated space 
for each lesson; n = 4; 2 teachers, 2 community aides). For example, one community aide 


suggested that future users of the program should “be willing to adapt the program to the kids — 
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consider their attention spans — and do what works for them without being obsessed with the 
suggested time frames;” one teacher provided feedback to “allow teachers to have more room for 
making the lessons unique.” Other responses encouraged the implementation of lessons by 
instructors other than the lead classroom teacher. For example, one teacher commented, “I would 
suggest having a special helper teacher implement the program. I think a lead classroom teacher 
cannot split the time as easily.” Another teacher reported, “I think the community aides would be 
a better position for this program due to turn over, absences of teachers and the ability to do it,” 
and reiterated to “please make sure there is enough staffing” when implementing the program. 
One community aide felt she did not learn enough about the types of children with whom she 
would work in a small group setting, and suggested, “Discuss small groups of children. I had 
three girls with focusing issues.” 
Discussion 

Given growing expectations that teachers support young children’s emergent literacy 
development and provide small-group literacy instruction, it is critical to examine how such 
practices are implemented within classrooms. The current study adds to both the literacy 
intervention and implementation literatures in applying a multi-dimensional framework (Durlak, 
2010; Durlak & DuPre, 2008) to describe, in detail, the implementation of one small-group 
emergent literacy intervention when implemented by two types of instructors. Importantly, this 
intervention is currently marketed to and used by preschool and early childhood programs in 
several states, including Florida, Delaware, and Virginia, and the instructors in this study 
received what would be typically provided to anyone who adopted the intervention (1.e., lessons, 
materials, and standard 2-day professional development). The results reconfirm implementation, 


including facilitators and challenges, as a key factor in successfully enacting research-based 
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practices within authentic classroom settings. Our major findings, discussed below, have 
important implications for intervention development, use, and scalability. 

By and large, instructors implemented NBS! such that it was delivered with high quality, 
afforded participant responsiveness, aligned with the intended lesson duration, and met the 
developers’ adherence criterion for the majority of small groups. These results support NBS! as 
an intervention that can be implemented in authentic preschool classrooms and substantiate many 
of the features integrated into NBS! to make it easy and engaging to use. The latter is also 
affirmed in instructors’ comments about the intervention (e.g., finding the scripted and routinized 
lessons, multisensory activities, and provision of high-quality, engaging books and materials 
helpful for implementation) and their overall positive perceptions concerning social validity. 
These design features may not only be helpful for non-teachers (Wasik, 1998), but also may be 
helpful for classroom teachers and promote scalability (Weiland, McCormick, Mattera, Maier, & 
Morris, 2018). Notably, instructors’ desires to modify the intervention to better fit the contexts 
in which they were working is unsurprising, as such adaptation is not atypical when 
implementing in authentic classroom settings (Durlak, 2010; Dusenbury et al., 2003). 

The variability in instructors’ adherence and the number of lessons implemented, 
however, is noteworthy, especially given evidence that these implementation dimensions are 
related to children’s learning (e.g., Bailet et al., 2009; Zucker et al., 2013). Average adherence 
was 76%, with a range of 48% to 97%, and 36% of small groups did not experience lessons that 
met the 75% adherence criterion. Only five small groups experienced all 20 lessons. The 
average number of lessons implemented was 12 of 20 (62%), with a wide range for both teachers 
and community aides. Although levels of implementation necessary for NBS! to achieve 


impacts on children’s learning remains an open question, these adherence and dosage levels are 
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substantially lower than those exhibited by NBS! instructors in prior efficacy work (97% 
adherence and 89% of lessons completed; Zettler-Greeley et al., 2018). This is not altogether 
surprising given that prior instructors were all directly hired, trained, and supervised by the NBS! 
developers. Evidence in other fields suggests that implementation under routine conditions by 
intended end users is often lower than implementation in highly controlled circumstances 
(Dusenbury et al., 2003). The few other studies examining teacher implementation of small- 
group emergent literacy interventions have reported somewhat higher levels of adherence and/or 
dosage but similarly wide variability (e.g., average adherence of 84% with a range of 46% to 
100%, average of 81% of lessons completed with a range of 53% to 100%, Goldstein et al., 
2017; average adherence of 85% with a range of 60% to 90%, Gonzalez et al., 2011; see also 
Pollard-Durodola et al., 2016). The higher implementation may be due to differences in the 
specificity with which adherence was measured, the intervention itself, or the sample. For 
example, the interventions in these studies tended to focus on fewer emergent literacy skills (e.g., 
phonological awareness and alphabet knowledge) or emphasize one specific instructional method 
(e.g., Shared book reading). With respect to the sample, in particular, instructors in studies by 
Gonzalez and colleagues (Gonzalez et al., 2011; Pollard-Durodola et al., 2016) consisted 
exclusively of teachers working within public school districts, the vast majority of whom held at 
least a bachelor’s degree and teaching certifications, and all of whom had regular classroom 
aides. 

This is in stark contrast to the current sample of teachers charged with implementing 
NBS!, who may be more representative of the early childhood workforce (National Survey of 
Early Care and Education, 2013; Rhodes & Huston, 2012): None of the teachers were public 


school employees, the vast majority did not hold teaching licenses, education levels were more 
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variable, and support staff was less available. Indeed, a majority of teachers in the current study 
cited lack of staff support as a barrier to implementation, given the small-group context. All of 
these ecological factors (e.g. instructors’ backgrounds, systems in which they work, and supports 
available in those systems) may account for differences in implementation (Durlak & DuPre, 
2008). For instance, instructors’ educational backgrounds and prior experiences with children, 
whether classroom teachers or community aides, may influence how they interact with children 
and manage behavior; in turn, this may affect the instructional quality with which interventions 
are implemented. The ecology of the classroom itself, in terms of classroom composition and 
organization, could also influence implementation. In the current study, we note that classrooms 
assigned to the community aide-implementation condition tended to have overall better 
classroom organization; this may have facilitated community aides’ abilities to work with small 
groups of children and provide greater dosage. Notably, factors outside the classroom such as 
teacher (or community aide) pay or policies governing teacher:child ratios may affect classroom 
staffing, capacity for providing small-group intervention, and thus implementation. These 
factors are important to consider further as they may relate to variability in implementation. 

The variability in adherence and dosage for instructors in both conditions argues for 
better supporting these aspects of implementation. In addition to staff support, comments 
pertaining to facilitators, challenges, and adaptations consistently referenced time, classroom, 
and behavior management. These comments validate prior conjectures regarding factors that 
may help or hinder small-group instruction in preschool classrooms (Wasik, 2008; Zucker et al., 
2013) and align with previous empirical findings that small groups are more likely to be used in 
the presence of better classroom management and adult:child ratios (Farley et al., 2017). 


Importantly, the comments imply a need for more opportunities to strengthen classroom or group 
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management skills. This may be equally important for community aides or other non-teachers 
who serve as instructors as for preschool teachers, given that neither may have completed formal 
teacher preparation programs. Relatedly, although many teachers referenced difficulties in 
“leaving” their classrooms, there was no requirement to conduct the intervention outside the 
classroom. This further suggests that small-group instruction, and the classroom routines it 
requires, was not common practice in these classrooms. This was reiterated in the comments of 
some teachers and community aides, who indicated a need to know more about small-group 
instruction. The standard, publisher-provided professional development that accompanies NBS! 
only briefly touched on the benefits of small-group instruction and did not delve deeply into 
strategies for managing small groups; professional development associated with small-group 
emergent literacy interventions should incorporate more opportunities to learn about small-group 
instruction, including strategies for successfully enacting and managing such instruction. 
Additional training in small group instruction and management may help instructors, in 
both teacher-implemented and community-implemented contexts, to reach higher levels of 
implementation (cf. Zucker et al., 2013), and this should be tested empirically in future research. 
Moreover, such supports may help to address the implementation tradeoff identified in the 
current study. Namely, teachers tended to have greater adherence and child responsiveness but 
implemented fewer lessons, whereas community aides implemented more lessons but with lower 
adherence and fewer children responding to instruction. Community aides, who may not be as 
familiar with emergent literacy concepts or instructional techniques, might benefit from 
additional training on intervention content (as might teachers), as well as methods to boost 
adherence and children’s responsiveness during lessons. Our findings suggest that leveraging 


community members as intervention instructors may be a viable alternative to teacher 
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implementation, assuming that adequate levels of adherence can be attained. In addition to 
overcoming staff support issues and potentially providing higher intervention dosage, 
community implementation may have advantages in efficiency: A single community 
organization may be able to serve children across multiple early childhood centers and, in cases 
of child mobility, follow children to continue providing intervention. Instructors may develop 
greater facility and expertise with the intervention if they are working with multiple small 
groups, and intervention would not necessarily stop during instances of teacher turnover, which 
is particularly high in the preschool sector (Rhodes & Huston, 2012). This option, in addition to 
teacher implementation, offers flexibility that might enhance use and scalability of NBS! or 
similar small-group emergent literacy interventions (Durlak, 2010). 

Our findings concerning facilitators and challenges to using small-group emergent 
literacy interventions, along with implications for implementation by teachers versus community 
aides, hopefully will inform the continued development of NBS! and other interventions. We 
acknowledge, however, that we examined implementation of only this particular intervention, 
and findings may not generalize to other interventions, samples, community partnerships, or 
contexts. We also acknowledge that a larger sample size may have resulted in greater statistical 
power to detect any differences between teacher-implemented and community aide-implemented 
conditions, although we were careful to consider effect sizes and practical significance in 
addition to statistical significance. We note that our findings pertain only to teachers’ first year 
of implementation and that implementation may improve over time (Clements, Sarama, Wolfe, 
& Spitler, 2015). Resource constraints prevented us from collecting follow-up data from 
participating teachers, and the small sample of community aides prevents us from analyzing 


improvement over time in any meaningful way; examining this is an important direction for 


IMPLEMENTATION OF SMALL-GROUP INTERVENTION 36 


future research. Furthermore, we note that individual teachers agreed to participate in this 
research project and NBS! was therefore not being used across all classrooms at a given center or 
school. Organization-wide adoption might affect implementation (Durlak & DuPre, 2008), 
particularly if this resulted in structural changes that addressed staffing and other challenges. 
Finally, we focused exclusively on implementation in this study. We did not address how 
engaging with the NBS! professional development and intervention may have affected teacher 
and community aide outcomes, such as content knowledge, pedagogical techniques, attitudes 
towards early learning and instruction, or desire to seek additional qualifications to support early 
literacy development. Such outcomes may influence whether teachers and community aides 
continue working in the early childhood sector. We also did not address whether NBS!, as 
implemented by teachers and community aides, affected children’s emergent literacy learning 
and leave this to future work. A critical next step will be to link levels of implementation to 
children’s outcomes (e.g., threshold or dose-response models) and to identify the relative 
contributions of particular aspects of implementation (e.g., adherence, quality) as these can refine 
the NBS! logic model. Such investigations of implementation are important as we strive to enact 
developmentally appropriate, research-based practices in authentic educational settings. The 
success of any intervention depends on how it is implemented in practice (Durlak, 2010) and the 
concept of “research-based” should also include implementation research as this informs 
practices that can be realistically and appropriately used in classrooms. Our findings identify 
potential issues and challenges that may now be proactively addressed when designing and using 
small-group emergent literacy interventions to improve implementation and, ultimately, 


intervention efficacy. 
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Table 1 
Characteristics, by NBS! Condition, for Participating Preschool Classrooms 


Classroom Condition * 


Community aide- 


Teacher-implemented implemented classrooms 
classrooms (n = 17) (n= 19) 
Classroom characteristics n % n % 
Female classroom teacher 17. 100.00 17 89.47 
Black classroom teacher 7 41.18 10 52.63 
Hispanic/Latinx classroom teacher 0 0.00 1 5.26 
Licensed classroom teacher 2 ' W176 3 15.79 
Teachers’ highest degree earned 
High school diploma 1 5.88 3. 15.79 
Associates 5 29.41 6 31.58 
Bachelors 10 58.82 7 36.84 
Masters 1 5.88 1 5.26 
Center-based 13. 76.47 16 84.21 
Public school-based 3 T7163 2 10.53 
Urban location 15 88.24 14 73.68 
Affiliated with Head Start 7 41.18 6 31.58 
Accepts public subsidies 9 52.94 9 47.37 
Day length 
Full 14 = 82.35 10 52.63 
Half 1 5.88 3 15.79 
Both half and full day 2 11.76 4 21.05 
Creative Curriculum 15 88.24 14 73.68 
M SD Range M SD Range 
Years of classroom teacher’s L133 8.11 0-28 12.53 8.78 0-30 
preschool teaching experience 
Class size 14.18 3.70 7-18 13.53 440 7-20 
NBS! small group size 3.41 1.06 1-4 2.79 1.08 1-4 
Teacher:child ratio? 1:6.27 2.27 4-13 1:6.41 2.90 2-14 
Children with IEPs*° 1.29 0.76 1-3 1.40 0.55 1-2 
Children learning multiple 
languages 4.44 321:; W=8 2.67 1.88 1-6 
Classroom organization® 4.31 0.76 2.5— 4.86 069 3.2- 
5.3 6.0 
Classroom instructional support® 2.06 0.43 1.3- 2.28 0.77 1.2- 
2.8 4.0 


Note. Percentages may not total 100% due to unreported values or rounding. We statistically 
compared classrooms on all characteristics using Chi-square test, Fisher’s Exact Test, or one- 
way ANOVA as appropriate to the categorical or continuous data. By chance, classrooms 
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randomly assigned to the community aide-implemented condition had significantly higher 
classroom organization scores, F(1,31) = 4.78, p = .036; we detected no other differences 
between conditions. 

“Characteristics refer to the classrooms participating in the study and the lead or co-lead teacher 
staffing those classrooms; this lead or co-lead teacher also implemented the intervention in the 
teacher-implemented condition. Characteristics do not refer to community aides; the seven 
community aides serving these classrooms are described in text in the Participants section of the 
Method. Teachers included all lead, co-lead, and assistant teachers as well as teacher aides 
regularly in the classroom; SD and range refer to the numbers of children per teacher. “In the 7 
teacher-implemented classrooms and 5 community aide-implemented classrooms serving this 
population. “In the 9 teacher-implemented classrooms and 12 community aide-implemented 
classrooms serving this population. “Organization and instruction support of the classroom as a 
whole (not intervention sessions) as measured using the Classroom Assessment Scoring System 
(CLASS; Pianta, La Paro, & Hamre, 2006) on which | represents low quality and 7 represents 
high quality. Research staff coded CLASS from 100 min videotaped classroom observations 
collected mid-year on a day considered representative of typical instruction. As aligned with 
recommendations by Mashburn, Meyer, Allen, and Pianta (2014), two 20 min cycles were 
randomly selected and coded in a random order; double coding of 15% of cycles indicated 93% 
agreement between coders. 
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Table 2 
Implementation Facilitators and Challenges Reported by Teachers and Community Aides 


Teacher Community Aide 


Facilitator Challenge — Facilitator Challenge 
Overarching code/subcode n n n n 


Lesson structure 
Provision of materials and 13 7 
multisensory activities 
Scripted lessons 1 6 
Repetition 4 


Ww 
Ww 


General implementation guidelines 6 
Difficulty of lesson content 
Small group instruction 
Child lack of interest/bored 
Engagement 
Interactive nature of lessons 2 6 
Child interest/learning 3 2 
Materials to involve parents 1 2 
Time management 


BRWN WN 
NO 


Practice and preparation ahead of time 4 3 

Patience and flexibility 1 2 

Incorporating lessons into daily routine 10 5 

Limited time/inability to complete 6 7 

lessons 

Lesson length (implementation and 6 1 

preparation) 

Make-up lessons 3 3 
Classroom management 

Small group instruction (e.g. one-on-one) 1 

Lack of staff support/coverage 6 

Child/teacher absenteeism 6 ri 

Managing small groups 3 


— 


Scheduling with teachers/centers/schools 4 
Behavior management 
Small group instruction 1 
(e.g. close relationships) 
Behavior issues during small groups 1 5 
(e.g. redirection) 
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Table 3 
Implementation Results, Overall and by Condition 
Teacher-implemented Community aide- 
small groups implemented small groups 
Overall (n = 36°) (n = 17*) (n = 19) Comparison? 
M SD Range M _ SD Range M SD Range t-value p-value d 

Adherence 

Key lesson elements 76 12 48-97 81.12) 48-.97 73, 12 48-92 157  .131 0.70 

implemented (proportion) 
Dosage 

Lesson duration’ (min) 42.50 13.87 20.45—-76.49 43.14 15.17 20.45-69.86 41.95 13.08 21.08-76.49 —-0.16 ~—- 876 -0.07 

Number of lessons 12.44 6.13 0-20 8.88 6.51 0-20 15.63 3.56 5-20 -3.05 .005 -1.17 

implemented (20 max) 
Quality of Delivery 


Quality of Intervention 2.16 0.32 1.53-2.75 2.19 0.31 1.81-2.75 213° 0.34, 153-2:56 0.62 .541 0.26 
Delivery subscale (0 to 3 
rating) 

Participant Responsiveness 
Child attention/ 95.08 ~—.67—1.00 94 10 .67—1.00 96 05 .83-1.00 -0.49 629 -0.20 
participation (proportion) 
Quality of Intervention 2.42 0.35  1.70-3.00 2.51 0.36 1.75-3.00 2.34 0.32 1.70-2.90 144 165 0.51 
Receipt subscale (0 to 3 
rating) 


Social Validity* 
Usage Rating Profile- 3.68 0.67 2.31-4.62 3.80 0.71 2.31-4.62 3.40 0.50 2.92-4.25 0.54 595 0.57 
Intervention scale (0 to 5 
rating) 


Note. *One teacher did not withdraw but did not implement any lessons; she thus did not contribute adherence, lesson duration, quality 
of delivery, or participant responsiveness data. Statistical comparison between the two conditions, with community aide-implemented 
model as the reference condition; degrees of freedom = 21 for all but number of lessons and social validity, when = 22. “Because all 
lessons were split into two 20-30 min sessions, the duration of these two sessions were summed to derive lesson duration. “Unit is 
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instructor (17 teachers, 7 community aides), as community aides reported on social validity once per year of implementation and not 
separately for each small group. 


Table 4 

Correlations Among Impl ementation Measures 

Measure 1 2 3 4 5 6 7 
Adherence 


1. Key lesson elements = 
implemented (proportion) 
Dosage 
2. Lesson duration (min) — .06 = 
3. Number of lessons -.08 30 = 
implemented (20 max) 
Quality of Delivery 
4. Quality of Intervention .77** — -.13 -.11 ce 
Delivery subscale (0 to 3 
rating) 
Participant Responsiveness 
5. Child attention/ .36* -.01 aa A6** = 
participation (proportion) 
6. Quality of Intervention .13 -.05 -.33 25 he fod = 
Receipt subscale (0 to 3 
rating) 


Social Validity 
7. Usage Rating Profile- -.08 -.17 .08 -.18 -.28 -.16 Z 
Intervention scale (0 to 5 
rating) 

7p 09." tp Ol. 
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Figure 1. Nemours BrightStart! logic model. 
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